The Annals of Applied Probability 
2009, Vol. 19, No. 4, 1347-1368 
DOI: 10.1214/08-AAP577 

© Institute of Mathematical Statistics, 2009 



TIME AVERAGES, RECURRENCE AND TRANSIENCE IN THE 
STOCHASTIC REPLICATOR DYNAMICS 

By Josef Hofbauer and Lorens A. Imhof 

University of Vienna and Bonn University 

We investigate the long-run behavior of a stochastic replicator 
process, which describes game dynamics for a symmetric two-player 
game under aggregate shocks. We establish an averaging principle 
that relates time averages of the process and Nash equilibria of a 
suitably modified game. Furthermore, a sufficient condition for tran- 
sience is given in terms of mixed equilibria and definiteness of the 
payoff matrix. We also present necessary and sufficient conditions for 
stochastic stability of pure equilibria. 

1. Introduction. The present paper deals with a stochastic variant of 
the continuous-time rephcator dynamics. We begin with a brief review of 
the deterministic model. For a comprehensive discussion, see Hofbauer and 
Sigmund (1998), Weibull (1995) and the recent surveys by Hofbauer and Sig- 
mund (2003), Nowak and Sigmund (2004) and Sandholm (2007). Consider 
a symmetric two-player game with n pure strategies, l,...,n, and nxn 
payoff matrix A = (aij). Thus both players have the same set of strategies 
and for either player, aij is the payoff from using strategy i if the oppo- 
nent uses strategy j. There is no symmetry or skew-symmetry assumption 
on A. The replicator dynamics describes how the proportions of strategies 
in a population evolve. Consider a large population where every member 
is programmed to play one pure strategy. Let Qi^) denote the size of the 

subpopulation of i-players at time t, and let ^j(t) =Ci(*)/[Ci(0 H ^Cn(t)] 

denote its proportion. If the population is in state ${t) = (^i(t), . . . ,^n{t))'^ , 
then, under random matching, {A^{t)}i is the expected payoff to individuals 
playing i. Suppose that this payoff represents the per capita growth rate in 
the ith subpopulation. Thus 

(1.1) Ci = Q{A^}i, i = l,...,n. 
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This yields the deterministic replicator dynamics of Taylor and Jonker (1978) 

(1.2) ^. = ^^[{A^}^-eA^], i = l,...,n. 

The observation that biological processes modeled by replicator dynamics 
are inherently stochastic in nature led Foster and Young (1990) to introduce 
a replicator model based on a stochastic differential equation, and it turned 
out that even small stochastic effects can qualitatively change the asymp- 
totic behavior. The present paper studies a stochastic model of Fudenberg 
and Harris (1992) which is similar to that of Foster and Young but has a 
more natural boundary behavior. Following Fudenberg and Harris, we intro- 
duce random perturbations to the payoffs modeled by independent Gaussian 
white noises with intensities af,. . .,a^. Instead of (1.1) we consider 

(1.3) dZi{t) = Zi{t)[{AX{t)},dt + aidWi{t)], i = l,...,n, 
where 

X = {Xi ,...,Xn)'^ = — — ■ -— {Zi,...,Zn)^ 

Zi + \- An 

and {Wi, . . . , Wn)"^ = is an n-dimensional Brownian motion. The evolu- 
tion of the population state X{t) is then given by the stochastic replicator 
dynamics 

(1.4) dX{t)=h{Xit))dt + C{Xit))dW{t), X(0)=xo, 
where 

b(x) = [diag(xi, . . . ,Xn) - xx^][A - diag((Ti, . . . ,cr^)]x 

and 

C(x) = [diag(xi, . . . , j;„) - xx^] diag(o-i, . . . , cj„) 

for X € A = {y G [0, 1]" :yi + • — \-yn = l}i and xq is an initial value in int A. 

Fudenberg and Harris give a complete analysis of the asymptotic behav- 
ior for games with two pure strategies. Further papers that study the case 
n = 2 include Saito (1997), Amir and Berninghaus (1998), Corradi and Sarin 
(2000) and Beggs (2002). Most of the analysis for the case n = 2 relies on 
tools specific to one-dimensional diffusions. Few papers treat the general case 
n>2. Cabrales (2000) considers extinction of dominated strategies, Imhof 
(2005a) examines the long-run behavior in the presence of evolutionarily 
stable strategies and Khasminskii and Potsepun (2006) analyze a related 
stochastic replicator model with Stratonovich type random perturbations. 
Benai'm, Hofbauer and Sandholm (2008) study the relation between recur- 
rence and permanence. Imhof (2008) analyzes stochastic dynamics for games 
that describe multiple-trial conflicts. Discrete stochastic replicator processes 
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have recently been investigated by Schreiber (2001) and Benai'm, Schreiber 
and Tarres (2004). 

The aim of the present paper is to provide further insight into the long- 
run behavior of the stochastic process {X{t)} and in particular of its time 
averages Jq X[t) dt in the general case n > 2. It turns out that a crucial 
role is played by the modified payoff matrix 

(1.5) A = {ciij)^j—i, dij = ttij — ^(Tj . 

In contrast to the deterministic solution ^{t), the stochastic process {X(t)} 
cannot converge to an interior point of A. In fact, the only points to which 
{X{t)} can converge with positive probability are the vertices of A, which 
correspond to populations consisting of one single type of players. This is one 
reason for our interest in the analysis of the time averages Jq X{t) dt, 
even in cases where (1.2) has a globally asymptotically stable equilibrium. 
In Section 2 we study limit points of the time averages and relate them to 
Nash equilibria and correlated equilibria of the modified game A. 

Section 3 deals with recurrence. As an immediate consequence of the 
results in Section 2, we obtain the following averaging principle: if {X{t)} 
is positive recurrent, then A has a unique interior Nash equilibrium and the 
time averages converge almost surely to this equilibrium. Furthermore, we 
consider a certain class of games that includes all zero-sum games, and give 
an explicit expression of the invariant densities for the transition probability 
functions of the corresponding replicator processes. 

In Section 4 we derive an exclusion principle: if there does not exist a 
strategy against which every opponent obtains the same payoff under A, 
then {X{t)} is transient. It is also shown that the process is transient if 
there does exist a mixed strategy with this property and A is conditionally 
positive definite, that is, y-^^y > for every y 7^ with yi + ■ ■ ■ + Un = 0. 
Moreover, we investigate the relation between pure Nash equilibria of A and 
stochastic stability of the corresponding vertices of A. Thus we obtain three 
sufficient conditions for transience of the replicator process. Section 5 relates 
our results to some of the central results for the deterministic replicator 
dynamics, including the folk theorem of evolutionary game theory. 

If we introduce random perturbations into (1.1) using Stratonovich inte- 
grals instead of Ito integrals, we obtain 

(1.6) dzf )(i) = Z^,'\t){AX^'\t)}^ dt + aiZi'\t) odWiit), i = 1, . . . ,n, 

where X^^) = (z|^^ + ■■■ + Z^n^Y^Z^'^ See Turelh (1977) for a compari- 
son of Ito and Stratonovich equations from a biological point of view. The 
Stratonovich equation (1.6) is equivalent to the Ito equation 

dZ^^\t) = Z^\t)\{AX^'\t)-\^ + \al\ dt + cT.zf ^(t) dWi{t), i = l,...,n. 
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Thus {X^'Ht)} 

can be regarded as a solution to the Ito stochastic rephcator 
equation of Fudenberg and Harris for the payoff matrix A^^^ with entries 
a\j^ = aij + ^af. Consequently, all the results in the present paper carry 
over to the Stratonovich solution {X^^\t)} when A is replaced by A^''\ For 
instance, Theorem 3.1(a) yields that if {X^'^^t)} is positive recurrent, then 
the time averages Jq X^^\t) dt converge to an interior Nash equilibrium 
of A. Note that modifying A^^^ as in (1.5) leads to the matrix A^'^^ = {af^ — 
^(T?), which is just the original payoff matrix A. Theorem 4.11(a) shows that 
every strict Nash equilibrium of A is stochastically asymptotically stable for 
{X('')(t)}.This improves a recent result of Khasminskii and Potsepun (2006); 
see Section 5. 



2. Time averages. Throughout we assume that {X[t)} = {X{t,uj)} is 
a strong solution to (1.4) with initial value xq G int A. Then {X{t)} is a 
Markov process and it is not difficult to see that, a.s., X{t) stays in int A for 
all t > 0. Note that if some strategies were not present initially, they would 
never occur, so that our results would apply to the stochastic dynamics in 
the interior of the corresponding face of A. 

We first prove a result that relates time averages of X[t) and Nash equi- 
libria of A. The connection with space averages will be considered in Section 
3. Recall that p G A is a Nash equilibrium of A if 

p^Jip > q^Ap for aU q € A. 

The equilibrium is said to be strict if the inequality is strict for all q 7^ p. 



Theorem 2.1. (a) For almost every uj for which 



1 

T 







T 

X{t,uj)dt 



converges as T ^ 00, the limit is a Nash equilibrium of A. 

(b) The following implication holds for almost all lo. If the limit 

1 fTkH 

y = y (^^) = lim T^^TT / -^(*' ^) dt 

k^oo lk[iO) Jo 

exists for a sequence of times Tk{io) with 

(2.1) Tk{uj)/ 00 and — — > 0, z = l,...,n, 

then y is a Nash equilibrium of A with {^y}i = • • • = {Ay}„. 
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Proof, (a) Note first that from (1.3) and Ito's formula, 

(2.2) dlogZ^it) = [{AX{t)}i - ^af]dt + aidWi{t), 
so that 

dlogXi{t) - dlogXj{t) 

(2.3) =dlogZi{t)-d\ogZj{t) 

= [{AX{t)}i - {AX{t)}j] dt + ai dW,{t) - Gj dWj{t). 

The event fio = {limf^oo ^jifyl^ — > for j = 1, . . . , n} has probability 1. Let 
ijjQ G r^o be such that y /q X(t,a;o) dt converges to y = y(u;o)) say. Clearly, 
y G A. If > 0, then there exist times = Tfc(wo) such that < Ti < T2 < 
• • • , Tfc ^ 00 and Xi{Jk-.^o) > \yi for all k. Thus \\.ogXi{Tk,iOQ)\/Tk 0. As 
OJQ G r^Oi it now follows by (2.3) that for every j, 

{Ay}i - {Ay}j = lim — / {AX{t,ujQ)}i - {AX{t,uJo)}j dt 

fc^oo Ik JO 

^ ^.^ log (Tfc , cjq) - log (Tfc , cjq) ^ ^ 

fc— >oo T/j 

This shows that y is a Nash equilibrium of A. 
(b) The proof is similar to that of (a). □ 

For every fixed lo consider the set of limit points {pij) = {pij{u))) of the 
time averages of the products Xi{t,uj)Xj{t,ijj) counting the encounters of 
plays of strategies i against j, that is, for some sequence of times Tk{uj), 

(2.4) rfc(a;)^oo, pij = lim / Xi{t,uj)Xj{t,uj) dt 

k^oo lk(iO) Jo 

Let Pi = J2]=iPij = linifc^oo /o^'"^'^'* ^iit, w) dt be its marginals. 

Theorem 2.2. With probability 1, each limit point (pij) satisfies 

1 " 

(2-5) iAp)i < dijPij + 2 II "^jiPj - Pjj) 

for all I, and equality holds for at least one I. 

In the deterministic case = for all i this result was obtained in Hof- 
bauer (2005). The corresponding inequalities state that the correlated distri- 
bution (pij) satisfies the exact marginal best response property of Fudenberg 
and Levine (1995), or is an element of the Hannan set as defined by Hart 
(2005), or a coarse correlated equilibrium in the terminology of Young (2004). 
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Proof of Theorem 2.2. From (1.3) we get for 5 = Zi H h Z„ 



dS = S 



Together with dlogXi = dlogZi — dlogS and (2.2) we obtain 



dlogXi 



(2.6) 



1 " 

{AX)i - X^AX - - E ^1^. (1 - 



3=1 



dt + ai dWi 



-^ajXjdWj. 



Integrating (2.6) gives 

logX;(r)-logXz(0) 1 



T 



{AXit))idt-J2 



1 



(2.7) 



with 



Xi{t)Xj{t)dt 



n{T) = ai 



Wi{T) 



Since Xj{t) is nonanticipative and |Xj(t)| < 1, it follows from Friedman 
[(1975), Corollary 4.6, page 77] that 



Consequently, 
(2.8) 



hm ^ [ Xj{t)dW,{t) = a.s. 

->oo I Jo 



lim r;(T) = a.s. 

T — ^oo 



Since logX;(T) < 0, every limit point of the left-hand side of (2.7) is nonpos- 
itive and hence the inequalities (2.5) follow. Now looking at one limit point 
{pij) arising for the sequence — > oo, let (after possibly refining the time 
sequence) x = lim^^.^oo ^(^fc) and choose I s.t. x/ > 0. Then the left-hand 
side in (2.7) goes to and there is equality in (2.5). □ 

Remark 2.3. For almost every uj for which the sequence of times Tk{uj) 
satisfies (2.1), the corresponding limit matrix (pij) = {pij{uj)) satisfies equal- 
ity in (2.5) for all /. 
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3. Recurrence. If the w-limit of the orbit of ^{t) given by the determin- 
istic equation (1.2) is contained in int A and A has a unique interior equihb- 
rium, then the time averages y /g ^{t)dt converge to that equihbrium; see 
Schuster et al. (1981) or Hofbauer and Sigmund (1998), Theorem 7.6.4. The 
fohowing theorem contains a stochastic counterpart and also estabhshes a 
connection with space averages. 

We write Pxo to indicate the probability computed under the condition 
X{0) = xq and E'xq to denote expectation with respect to Pxq- For x € A and 
e > let -Be(x) = {y G A : ||y — x|| < e}, where || • || denotes the Euclidean 
norm. The process {X{t)} is called recurrent if for every x € int A and every 
e>0, 

Pxu{^(7fe) £ for a sequence of finite random times 

increasing to infinity} = 1. 

See Bhattacharya (1978) for various characterizations of recurrence of diffu- 
sions. If {X{t)} is recurrent, its transition probability function has a unique 
(up to positive multiples) cr-finite invariant measure v on int A [Khas'minskii 
(I960)]. If z^(int A) < oo, {X{t)} is said to be positive recurrent. In this case, 
we will normalize v and consider the invariant distribution vr = [i/(int A)]~^i/. 
If {X{t)} is recurrent and i/(int A) = oo, {X{t)} is said to be null recurrent. 



Theorem 3.1 (Averaging principle), (a) If{X(t)} is positive recurrent 
with invariant distribution vr on int A, then y '■= J ydvr(y) is the unique 
interior Nash equilibrium of A and 

1 

lim — / X(t)dt = y a.s. 
T->oo T Jo 

(b) Let M = {y £ A: {Ay}i = ■■■ = {^y}„}. // {X{t)} is null recurrent, 
then TV 7^ 0. Moreover, if (Ta;)^-^ is a sequence of random times such that 

/Q i\ . , log^j(7fc) , 

(o.lj Ik / oo a.s. and — > (J a.s., i = l,...,n, 

Tk 

then, a.s., every accumulation point of the sequence 
^ X{t)dt, fc = l,2,..., 

belongs to the set M. 

Proof, (a) If {X{t)} is positive recurrent with invariant distribution 
vr on int A, then y := J y'ivr(y) G int A. Moreover, by the ergodic theorem 
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[Bhattacharya and Waymire (1990), page 623], limr_^oo lo ^{t) dt = y a.s. 
It follows from Theorem 2.1(a) that y is an equilibrium of A. 

It remains to show that y is the only interior equilibrium of A. If not, 
then there is a line of equilibria which implies the existence of a nonzero 
vector c G M" such that = and c'^A'k = holds for all x € A. This 
together with (2.6) implies 

n n 

(3.2) ^Q(ilogXi(t) = ^QCJ,dTyi(t). 

i=l 1=1 

Integrating gives 

i=l ' 1=1 

The right-hand side is a one-dimensional Brownian motion and hence null 
recurrent. This contradicts the assumption that the process {X{t)} is posi- 
tive recurrent in int A. 

(b) The assertion is an immediate consequence of Theorem 2.1(b). Note 
that null recurrence implies that there exists a sequence (T^) satisfying (3.1). 
□ 



Remark 3.2. (a) Generically, the set defined in Theorem 3.1(b) con- 
tains at most one element. Thus, if AA= {y}, then 

1 f'^" 

lim — / X{t) dt = y a.s. 

for every sequence of random times (T^) with (3.1). 

(b) Every interior Nash equilibrium of A belongs to J\f. Thus if {X{t)} is 
positive recurrent, then y € M, and as y is the only interior Nash equilib- 
rium, M = {y}. 

Theorem 3.3. If {X{t)} is positive recurrent with invariant distribu- 
tion TT on int A, then the time averages in (2.4) converge a.s. to pij := 
J yiUj d-K{y) . Therefore, a.s., there is a unique limit matrix, (pij), all its 
elements are positive, and there holds equality in (2.5) for all I. 

Proof. This follows again from the ergodic theorem. For the last state- 
ment, choose a time sequence — > oo with (3.1) and apply Remark 2.3. 

□ 

For n = 2 we can compute this limit matrix explicitly: Let A= {^^ ^ ^ 

and (Ji = o"2 = o" > 0. Then by Proposition 1 of Fudenberg and Harris (1992) 
or Corollary 3.10, {X{t)} is positive recurrent, so that, by Theorem 3.1(a), 
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Pi=P2 = ^- Hence (^p)i = | — Since there holds equahty in (2.5) and 
since J2jPij = Pi ^iid pi2 =P2i, we obtain 

(^P)l = {Pl2+P2l) - ^(^'ll +P22) =2pi2 - - 2pi2). 

It follows that pi2 = P21 = 4_|_2g-ii < \ and pu = P22 = \-Pi2> \- For large a 
the matrix (py) approaches ^ -^^2 ) ' which means that the process spends 
most of the time near the pure strategy states. 

Remark 3.4. If the process {X[t)} is null recurrent, then for almost all 
Lo there is at least one accumulation point {pij) = {pij{u))) for which equality 
holds for all I in (2.5). 

Corollary 3.5. Suppose A = —A^ is a zero-sum game. Then the pro- 
cess {X{t)} is not positive recurrent. 

Proof. If it were positive recurrent, then, by Theorem 3.1, Ap = for 
some p € int A, and Theorem 3.3 implies equality for all I in (2.5). Therefore 
J2]=i '^]iPj ~ Pjj) — 0- Since aj > we get pj = pjj which contradicts pij > 
for all i,j in Theorem 3.3. □ 

We next investigate invariant densities and recurrence properties of {X{t)} 
for payoff matrices that satisfy for some 7 G M the condition 

(3.3) aij + aji - an - ajj = ^ {af + cr|) for all i / j. 

If n = 2, (3.3) is always satisfied for a certain 7. For general n, the payoff 
matrices satisfying (3.3) can be characterized as those obtained by subtract- 
ing diagonal elements 7<7j/2 from the payoff matrix of a zero-sum game, and 
rescaling [i.e., adding multiples of 1 = (1, ... , 1)"^ to each column] or adding 
multiples of 1^ to each row. Note that condition (3.3) is equivalent to the 
condition obtained by replacing every aki in (3.3) by dki- 

Theorem 3.6. Let a S M" and set 7 = ai + • • • + a„. The transition 
probability function of {X{t)} has an invariant density (with respect to 
Lebesgue measure on int A ) given by 

n 
i=l 

if and only if one of the following two conditions (a) and (b) holds. 

(a) A satisfies (3.3) with 7 7^ —1 and {Act}i = • • • = {^q;}„ and if j = 0, 
then {Acxji = J^Li - ^ ELi (^faf- 
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(b) 7 = — 1 and for k = 1, . . . ,n, 

2 n n 

O I. 1 2 2 2 ~ - r» 

Y + 2 2^ '^i «i + "fcO-fc - Ofcfc - 2^ ajOjfc = 0. 

i=l i=l 

Proof. For simplicity, we work with the ]R"~^-valued process Y{t) = 
'i/{X{t)), where ^'(xi, . . . , x^) = {log{xi/xn), ■ ■ ■ ,log{xn-i/xn))'^ ■ Let de- 
note the kih column of the n x n identity matrix. We have from (2.3) 

dYj{t) = {{ej-enfA^-\Yit))}dt + ajdWjit)-andWn{t), 

j = l,...,n — 1. Let L denote the formal adjoint of the differential operator 
corresponding to {Y{t)}, that is, 

i,j=l i=l 

for / € C^(M"~^). A change of variables shows that HiLi ^t^"^ is an invariant 
density for the transition probability function of {X{t)} if and only if 

.(y) = ^-Pgrj/"-^-) , yeM'^- 

(1 + efi H hey"-i)^i=i"^ 

is an invariant density for the transition probability function of {Y{t)}. 
Set 

0(y) = l + e2'i + --- + e^"-^ 
Then for all i,j = l,...,n, j, 

dg{y) 9{y) 



dyi 0(y) 



ai(t>{y) -16^'], 

{[aiHy) - 7e^f + ley^ [e^' " Hy)]}, 



dyf (/>2(y) 
Hence 

/ \ f n—l n—1 

(3.4) Z9(y) = ^ ^ 5] PHe^^-+^' + ^ P^^e^'' + + E /^fc^''' 
where 



.l<fc</<n-l k=l k=l 



1 
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Pkl 



E 



2 2 
0"i 



■7 



1 



- akk - an + (7 + l)(aM + aik) - ^ai(ajfc + 5^; 

For A; 7^ Z, 

(3.5) Pkl- Pk- Pi = {l + '^) 



Furthermore, 
(3.6) pi 
where 



/3n = 



an + - akk - - ^(o"*; + f^i 



B = A 



T ~-,T 2sl'^ + 7diag(cj?, 



a — (ail, . • • , Onn) 

Condition (3.3) is equivalent to 
(3.7) B 
Now suppose n" 



i + la^ + -ls^. 



"=i Xj ' is an invariant density for the transition proba- 
biUty function of {X{t)}. Then Lg = 0; see Theorem 8.4 in Pinsky (1995), 
page 181. Hence, in view of (3.4), pk = for ah k and pki = for all k^l. 
Suppose first that 7 ^ — 1. It then follows by (3.5) that A satisfies (3.3). 
Moreover, by (3.6) and (3.7), 



1 



2 ^ 



BoL 



i=l 



a cx-\ — s ct 1 



A(x. 



This shows that Aa is proportional to 1, and if 7 = 0, then {74q:}i = 
J27=iauo.i — ^J2^=iO'faf. Thus condition (a) is satisfied. If 7 = —1, then 
condition (b) holds because Pk = for all k. 

Conversely, suppose condition (a) holds. Then, by (3.7), 



(3.8) 



Ba 



a^a + ■^s"' a ) 1 — Aa = pi 



for some p€R. Thus a^Ba = jp. By (3.7), B^ + B = 7diag(c7f,. . . ,<t^), 
and so 27/i = ol^{B'^ + B)cx = ^Yli=i o^f^^i- If 7 / 0, it follows that p = 
^ J27=i (^I'^i- If 7 = 0, then, according to condition (a), Aa = [{J2i=i oaian — 
^afai)]l, and it is obvious from (3.8) that again p = ^J27=i'^i^i- I* there- 
fore follows from (3.8) and (3.6) that Pk = for all k. Equations (3.3) and 
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(3.5) now imply that p^i = for all k^l. Consequently, by (3.4), Lg = 0. If 
condition (b) holds, then pk = for all k, and by (3.5), pki = for all k I. 
Thus, again, Lg = 0. It now follows by Theorem 8.5 in Pinsky [(1995), page 
182] that g is an invariant density, provided the diffusion corresponding to 
the solution to the generalized martingale problem for on M""^ does not 
explode, where f = -L{gf). We have 



9 

n— 1 ( n—1 



r- 



where 



1 ""^ r)2 ri-l f)f(v) 



and 



1 ^^(y) 76^' 
: a,; 



9{y) Vi 0(y) ' 

Thus the coefficients of are Lipschitz continuous and bounded so that 
the martingale problem for has a unique solution and explosions do not 
occur. □ 

Remark 3.7. Suppose A satisfies (3.3). Then A is conditionally negative 
[positive] definite if 7 > [<]0. Indeed, if yi + • ■ • + yn = 0, then, by (3.3), 

2y^yly = ^ yi{aij + aji - an - ajj)yj = -7 ^iVi- 

In particular, if (3.3) holds with 7 7^ 0, then there can be at most one a 
with ai + • • • + On = 7 that satisfies the equations {Aa}i = • • • = {^o;}n in 
condition (a) of Theorem 3.6. 

The density 0^=1 ^T~^ Theorem 3.6 is integrable if and only if > 
for all i. A random variable U with P{U G int A} = 1 which has a density 
that is proportional to 07=1^^'"^ said to have a Dirichlet distribution 
with parameter a = (ai, . . . , On)^, provided > for all i. If U has such a 
distribution and 7 = ai + • • • + On, then 

ai(7 - oti) 



(3.9) EC/ = 7-^0, Yw[U^] 



72(7 + 1) ■ 



Corollary 3.8. The process {X{t)} is positive recurrent and its tran- 
sition probability function has an invariant Dirichlet distribution with pa- 
rameter a if and only if A satisfies (3.3) with some 7 > and 7~^a is an 
interior Nash equilibrium of A. 
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Remark 3.9. If n = 2, then the condition that A satisfies (3.3) with 
some 7 > and A has an interior Nash equihbrium is equivalent to the in- 
equahties di2 > 0,22 and 021 > an. For this case, the stable coexistence case, 
Fudenberg and Harris (1992) already calculated the ergodic distribution of 
{X[t)} by means of the normalized speed measure. 

Corollary 3.10. Suppose A satisfies (3.3) for some ^ ^M.. Then{X{t)} 
is positive recurrent if and only if A has an interior Nash equilibrium and 
7>0. 

Proof. To prove the sufficiency of the condition suppose that x is an 
interior Nash equilibrium of A and 7 > 0. Set a = 7X. Then 7~^Q! is an inte- 
rior Nash equilibrium and so, by Corollary 3.8, {X{t)} is positive recurrent. 

To prove necessity, suppose {X(t)} is positive recurrent. Then, by Theo- 
rem 3.1(a), A has an interior Nash equilibrium x, say. Let a = 7X. If 7 7^ —1, 
then Theorem 3.6 shows that JliLi ^t'~^ invariant density, which must 
be integrable, so that 7 > 0. It remains to rule out that 7 = — 1. If7 = — 1, 
A would be conditionally positive definite by Remark 3.7, so that Theorem 
4.5 would yield that {X{t)} is transient. □ 

Remark 3.11. Corollaries 3.8 and 3.10 show that for games with (3.3) 
only Dirichlet distributions can be invariant distributions, and if {X{t)} is 
positive recurrent and cJi = • • • = an, then {X{t)} remains positive recurrent 
if 0"!, . . . , (Tn are replaced with Kai, . . . , kcj„ for any k > 0. As At \ 0, 7 /" c» 
and, in view of (3.9), the invariant distribution converges weakly to the unit 
mass concentrated at the Nash equilibrium oi A. As k y 00, the invariant 
distribution converges vaguely to the zero measure on int A. 

Corollary 3.12. The density Y\i'=iX~^ , x G int A, is invariant for the 
transition probability function of {X{t)} if and only if 

(3.10) aij + aji — an — Ojj = for all i,j. 
Moreover, if (3.10) holds and there exists (3 G M" such that 

(3.11) /3i + ... + /3„ = 0, {Ap}i = --- = {A/3}n and p^A^O, 

then nr=i another invariant density, where ai = cPi and c = 2{0^ A\i/ 

(J27=i^i(^i) 7^0- ^^^-^ case, {X{t)} is transient. 

Proof. The claimed equivalence is an immediate consequence of The- 
orem 3.6. If (3.10) and (3.11) hold, then 

n -j^ n n n 

^diiOi - - '^cffa^ = c^{aii - dii)(3i = c^(aii - au)(3i = {Aa}i, 

i=l 1=1 1=1 1=1 
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SO that niLi 2^^' invariant density by Theorem 3.6. Furthermore, 

(3.10) and (3.11) imply that = • • • = {/3^i|n, and so, as / 0, 

we have c / 0. Thus the invariant densities HILi ^7 nr=i ^f'"^ 
proportional. This implies that {X{t)} is transient; see Khas'minskii (1960) 
or Pinsky (1995), pages 148 and 181. □ 

Remark 3.13. If A is a zero-sum game, then (3.11) is equivalent to 
{A(3}i = ■■■ = {A(3}n + 0. 

4. Transience and stability of Nash equilibria. The deterministic repli- 
cator dynamics satisfies an important exclusion principle: if the replicator 
equation (1.2) has no interior rest point, that is, if there does not exist 
p G int A such that {^p}i = • • • = {^p}„, then every orbit converges to the 
boundary of A; see Hofbauer (1981) or Hofbauer and Sigmund (1998), The- 
orem 7.6.1. The following result for the stochastic replicator dynamics yields 
convergence to the boundary under a slightly stronger condition on A. 

The process {X(t)} is said to be transient if Py^f^{X(i) — > bd A} = 1. 

Theorem 4.1 (Exclusion principle). // there does not exist p G A such 
that {^p}i = • • • = {Ap}„, then {X{t)} is transient. 

Theorem 4. 1 is an immediate consequence of the averaging principle (The- 
orem 3.1) and the dichotomy between recurrence and transience; see, for ex- 
ample. Theorem 8.1 in Pinsky (1995), page 74. In the following we present 
an alternative proof. 

Proof of Theorem 4.1. If there is no p G A such that {Ap}i = • • • = 
{ylpln, then by a simple separation argument [see, e.g., Hofbauer (1981)] 
there exists a c G M" such that ci + • — h = and c^Ax. > for all x G A. 
By compactness there is 5 > such that c-^^x > 6 > for all x G A. Since 
ci + • ■ • + Cn = 0, (2.6) implies 

n n 
(4.1) J2''idlogXi{t)=C^AX{t)dt + Y,C^a^dWiit). 

i=l 1=1 

Integrating gives 

n X (t) " 

E log ITM ^ '^^ + E ^^<^^W,it), 
i=l ^''^^J i=l 

and hence J27=i logXj(t) oo a.s., as t — > cxo, which shows transience. □ 

Corollary 4.2. If strategy k is strictly dominated in the game A [i.e., 
there is a strategy q G A such that (Ax)^ < q-'^Jlx for all x G A/, then 
Xk{t)^0 a.s. 
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Proof. Choose c = q — in the previous proof which shows 

n 

^qi\ogXi{t) -\ogXk{t) ^ oo a.s. 

i=l 

Since the sum is bounded above, \ogXk{t) — oo a.s., or Xk{t) a.s. 

□ 

The following example shows that the weaker condition that A has no 
interior Nash equilibrium (or that strategy k is weakly dominated) is not 
sufficient for transience. 

Example 4.3. Suppose n = 2 and an = 021, 012 > 022- Then A does 
not have an interior Nash equilibrium. Using Feller's criterion [see, e.g., 
Proposition 5.22 in Karatzas and Shreve (1991), page 345] one may show 
that {X{t)} is recurrent. In view of Theorem 3.1(a), {X{t)} must be null 
recurrent. 

Remark 4.4. If A satisfies (3.10) and there exists (3 satisfying (3.11), 
then ^Si + • • • + /3„ = and {/3^i}i = • • • = {/3^i}„, / 0. That {X{t)} must 
then be transient can also be shown as in the alternative proof of Theorem 
4.1. 

For the second sufficient transience criterion recall that an n x n matrix 
B is said to be conditionally positive definite if By > for all y 7^ with 

yi H h yn = 0. Note that A is conditionally positive definite if and only if 

A is conditionally positive definite. 

Theorem 4.5. Suppose there exists p G M"\{ei, . . . ,e,„} such that {Ap}i = 
• • • = {Ap}n andpi-\ \-pn = 1. Suppose further that A (or A) is condition- 
ally positive definite. Then min{Xj(t) : i G 7+} — a.s., where /"^ = {i -.pi > 
0}. In particular, {X[t)} is transient. 

Proof. Set 

n 

ipipi.) = ^ pj log Xj, X G int A. 

By (2.6), 

(l){X{t)) = 4>{X{0)) + r h{X{s)) ds + R{t), 
Jo 
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where 

1 " 

hix) = p^ix - x^ix - - ^ a]xj{l - xj) 

3=1 

1 " 

= -(P - x)^i(p - x) - - ^ a'^jX.jil - Xj) 

3=1 

and 




Since A is conditionally positive definite, (p — x)^j4(p — x) > for all x G A, 
with equality if and only if x = p. Moreover, Yl^=i CTjXj^l — Xj) > for all 
x G A, with equality if and only if x G {ei, . . . , e„}. As p G M" \ {ei, . . . , e,„}, 
it follows that there exists some £ > such that 

/i(x) < —e for all x G int A. 



In view of (2.8), R(t)/t^O a.s. Consequently, a.s.. 




Thus min{Xi(t):iG/+}^0 a.s. □ 



Remark 4.6. If yl is conditionally positive definite and there exists 
p G M" \ A such that {j4p}i = • • • = {Ap}„ and pi H — ■ +Pn = l, then there 
cannot exist q G A with {^q}i = • • • = {ylq}„, and it follows already by 
Theorem 4.1 that X(t) converges to bdA a.s. Theorem 4.5 provides in this 
case a proper subset of bdA to which X{t) converges a.s. 

Remark 4.7. Theorem 4.5 complements the following stability result of 
Imhof (2005a): If A has an interior Nash equilibrium and A is conditionally 
negative definite, then {X{t)} is positive recurrent and spends most of the 
time in a small neighborhood of the equilibrium, provided cJi,...,cr„ are 
sufficiently small. Note that the transience result holds without restriction 
on the size of noise. 

Corollary 4.8. If A is conditionally positive definite and none of the 
columns of A is proportional to 1, then {X(t)} is transient. 

Proof. If there does not exist p G A with {^p}i = • • • = {^p}„, then 
Theorem 4.1 yields transience. Otherwise, there does exist such a p G A, 
but if none of the columns of A is proportional to 1, then p cannot belong 
to {ei, . . . ,e„}. Transience now follows from Theorem 4.5. □ 
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Remark 4.9. If one of the columns of A is proportional to 1, then 
{X(t)} cannot be positive recurrent. This follows from Theorem 3.1(a). 

Example 4.10. In the case n = 2 the assumptions of Theorem 4.5 are 
satisfied if an > 021 and 012 < 022- This means that the game A is bistable: 
both pure strategies are strict equilibria. 

A point p G A is said to be stochastically asymptotically stable [Arnold 
(1974), page 181] if for every open neighborhood [/ of p and every e > there 
is a neighborhood of p such that for every initial state x € y PI int A, 

Pjx(t) G U for aU t > 0, lim X(t) = p| > 1 - e. 

For every p G A \ {ei, . . . , e„} and every x G int A, P^{X{t) ^ p} = 0; see 
Imhof (2005a). Thus only the vertices of A can be stochastically asymptot- 
ically stable. 

Theorem 4.11. (a) Suppose strategy k is a strict Nash equilibrium in 
the game A, that is, 

akk >ajk + — ^ for all j ^ k. 

Then is stochastically asymptotically stable. 
(b) Suppose that for some x G int A, 



pJlimX{t)=ek}>0. 



Then strategy k must be a Nash equilibrium of A. Moreover, if there exists 
i^k with OLik = dkk, then there exists j ^ k such that dij < d^j. 

Proof, (a) Let L denote the second-order differential operator associ- 
ated with {X(t)}, that is, 

j=i ■> j,k=i J 

where 

fej(x) = Xj{ej - x)^[A - diag(o-i, . . . ,cr^,)]x, 

7,.(x) = E c,.(x)c,.(x), c,,(x) = I j'i^^^;;^ ] J l[ 

Consider the Lyapunov function 
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where r > will be chosen later. Clearly, 0(x) > for all x € A, and = 
if and only if x = e^. We will show that there exists a neighborhood V of 
such that for some c > 0, 

L(l){x) < -C(j){x) for all X G F n int A. 

This yields the assertion by Theorem 4 and Remark 2 in Gichman and 
Skorochod (1971), pages 314-315, or Theorem 4.1 in Has'minskii (1980), 
page 167. 

For all X G int A, 



L(/>(x) = r (ej - xf[A - diag(a?, . . . , al)]x + ^ 7,-,(x)x^" 



r-2 



: r 



E ^^ (e^- - ^)^^^ + ^ (1 - + E 

jj^k I \ i/=l 



,2 



1 " 1 



< r E ~ x)-^Jix + r maxjcrf , . . . , cr^}}. 

Since /c is a strict Nash equilibrium for A, there exists e > such that for all 
j 7^ A:, (efc — ej)-^JLefc > e. Thus there is a neighborhood V of such that 

(x — ej)'^Ax > ^ for all x G y and j ^ k. 

Now if r > is so small that r maxjo"^, . . . , cr^} < |e, then for all x G l^flint A 

e e\ re 



L<A(x)<r^x;:(^-i + -j=--<^(x). 

(b) If lim^^oo -'^(i,'^) = e/c, then the time averages ^ Jq X{t,io) dt con- 
verge to efc, too. It follows from Theorem 2.1 that strategy /c is a Nash 
equilibrium. For an indirect proof of the remaining part assume there ex- 
ists i ^ k such that dik = dkk and dij > dtj for all j . Since two-dimensional 
Brownian motion is recurrent, there is a sequence of random times {Ti)"^^ 
such that Ti ^ oo and, for every /, Wi{Ti) > and Wk(Ti) < 0. It follows 
that for every I, 

= r\AX{t)}, - {AX{t)}k dt + aiW,{Ti) - akWk{Ti) > 0, 
Jo 

which contradicts the assumption that P^{limt^oo X (t) = e^} > 0. □ 
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Remark 4.12. Theorem 4.11(a) improves Theorem 4.1 of Imhof (2005a), 
where the same stabihty assertion was proved under the stronger condition 
that 

Ofcfc > CLjk + crl for all j ^ k. 

Note that in view of Example 4.3 or Theorem 4.11(b), the stability condition 
in Theorem 4.11(a) cannot be weakened to 

akk >ajk + ^ ^ for all j / k. 

Remark 4.13. If n = 2, Theorem 4.11 can be strengthened. For n = 2, 
the following three assertions are equivalent. 

(a) Strategy /c is a strict Nash equilibrium of A. 

(b) The point is stochastically asymptotically stable. 

(c) For some x e int A, Px{limt^oo X(t) = ek} > 0. 

This is in contrast to the deterministic case where a pure strategy can be 
asymptotically stable without being a strict equilibrium. 

For the proof, it remains to show that (c) implies (a). Suppose P^{X{t) 
ei} > 0. By Theorem 4.11(b), an > 021, and if an = 021, then a22 < ai2. But 
if an = a2i and 022 < 0.12, then {X{t)} is null recurrent; see Example 4.3. 
This contradicts the assumption that Px{X(t) ei} > 0, and it follows that 
an > a2i. 



Corollary 4.14. If {2,3} and A is conditionally positive definite, 
then {X{t)} is transient. 

Proof for n = 3. If none of the columns of A is proportional to 1, then 
transience follows from Corollary 4.8. Suppose now that, for example, an = 
021 = 031 • Since A is conditionally positive definite, Kdu — aji — dij + djj for 
i^j. Hence ai2 < 022 and ais < 033. Moreover, < 022 - «32 - 023 + 033, so 
that a22 > 032 or 033 > 023. It follows that strategy 2 or strategy 3 is a strict 
Nash equilibrium of A. Thus, by Theorem 4.11, e2 or 63 is stochastically 
asymptotically stable and {X(t)} is transient. □ 



Example 4.15. Consider the rock-scissors-paper game with payoff ma- 
trix 



ai,a2 > 0. 





( » 


-ai 


02 




02 





-ai 




\-a\ 


02 
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Thus strategy 3 beats strategy 2, 2 beats 1 and 1 beats 3. If ai > 02, then 
for all y G \ {0} with yi + 1/2 + = 0, 

y^Ay = + yl + yf} > 0. 

Hence, according to Corollary 4.14, {X{t)} is transient. On the other hand, 
if oi < 02, {X{t)} is positive recurrent, provided (Ti,cr2,cT3 are sufficiently 
small; see Imhof (2005b). Moreover, if ai < 02 and o"i = (T2 = (T3 > 0, Corol- 
lary 3.8 yields that the invariant distribution is the Dirichlet distribution 
with parameter (02 — ai)/(3o"^)l. Finally, if ai = 02 and <7i = a"2 = C3 > 0, 
then Corollary 3.5 and 3.10 each imply that {X{t)} is not positive recurrent. 
To see that this follows from Corollary 3.5, note that the stochastic repli- 
cator dynamics remains unchanged if a constant is added to every entry in 
a column of A. In view of Corollary 3.12 we conjecture that {X{t)} is null 
recurrent in this case. 

Corollary 4.16. Suppose A satisfies (3.3) with some 7 G M. Then each 
of the following conditions is sufficient for transience of {X{t)}. 

(i) 7<0 and TiG {2,3}. 

(ii) There exists p G A \ {ei, . . . ,e.„} with {^p}i = • • ■ = {^p}i and 7 < 

0. 

(iii) There does not exist p G A with {A^p^i = • • • = {Ap}i. 

Proof. This follows from Remark 3.7, Corollary 4.14 and Theorems 4.1 
and 4.5. □ 

5. Related results. The aim of this section is to put our results into 
perspective by discussing the connection with corresponding results for the 
deterministic replicator dynamics. We concentrate on the following funda- 
mental properties of the deterministic model (1.2). 

(A) Every Nash equilibrium of A is a rest point. 

(B) Every strict Nash equilibrium of A is asymptotically stable. 

(C) Every stable rest point is a Nash equilibrium of A. 

(D) If an interior orbit converges, its limit is a Nash equilibrium of A. 

(E) Every evolutionarily stable strategy (ESS) of A is asymptotically stable. 

(F) If (1.2) is permanent, that is, if there exists a compact set K C int A 
such that for every initial state ^(0) = x G int A, ^(t) G K for t large 
enough, then there exists a unique interior rest point and for any initial 
state in int A, the time averages T~^jQ^{t)dt converge to that rest 
point. 
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Parts (A)-(D) constitute the folk theorem of evolutionary game theory. For 
proofs of all of these statements, see Hofbauer and Sigmund (1998). The 
stochastic replicator dynamics (1.4) has no absorbing points except for the 
vertices ei, . . . ,e„, so that part (A) is in marked contrast to the stochastic 
situation. Note that the converse of (A) holds neither for the deterministic 
nor for the stochastic replicator dynamics. 

Theorem 4.11(a) shows that part (B) essentially carries over to the stochas- 
tic case if the payoff matrix A is replaced by A. In contrast to what might 
be expected from the deterministic case, the assumption that a given strat- 
egy is a strict Nash equilibrium does not imply almost sure convergence 
to the equilibrium for suitable initial values. Indeed, if A describes a co- 
ordination game with two strategies, then for every initial state x € int A, 
Px{X{t) — >■ ei} < 1. Theorem 4.11(b) can be regarded as a weak analog of 
part (C). A sufficient condition for Py^{X{t) — > e^} = 1 for all x € int A is 
given in Imhof (2008). 

For the process {X^^\t)} determined by the Stratonovich stochastic dif- 
ferential equation (1.6), we obtain from Theorem 4.11(a) (applied to the 
corresponding Ito equation) that is stochastically asymptotically stable 
if Ofcfc > maxj^kajk- That the inequality 



is sufficient for stochastic asymptotic stability of has been shown by 
Khasminskii and Potsepun (2006). In view of Example 4.3, the sufficient 
condition a^k > niaxj^ytaj/t cannot be weakened to a^fc > maxj^f^ajk- 

Theorem 4.11(b) also yields the following analog of part (D). If for some 
x € int A and some p G A, Px{X{t) ^ p} > 0, then, by Theorem 4.3 of 
Imhof (2005a), p G {ei,...,e„}, so that, by Theorem 4.11(b), p is a Nash 
equilibrium of A. However, the natural stochastic analog of (D) is provided 
by Theorem 2.1(a). 

Concerning part (E) note that in Example 4.3, strategy ei is an ESS of 
A, which is not stochastically asymptotically stable. However, the existence 
of an interior ESS of A implies positive recurrence and that the invariant 
distribution puts most mass near the ESS, provided that ai, . . . , are small 
enough; see Imhof (2005a). Since the trajectories do not converge to the 
ESS nor to any other point, it is natural to study convergence of their 
time averages. By Theorem 3.1(a), the averages converge a.s. to the unique 
interior Nash equilibrium of A. 

Theorem 3.1(a) can also be seen as showing that an analog of the con- 
clusion of part (F) holds when {X{t)} is positive recurrent. The connection 
between permanence and recurrence is examined in Benai'm, Hofbauer and 
Sandholm (2008). 
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